CNBC Bureau Chief Jim Goldman was on the flight. He said the smoke became obvious as soon as the wheels left the ground and it was quickly followed by a rancid smell.
He said the plane returned immediately to the airport.
Goldman said a large force of emergency crews were on hand for the unplanned landing.
The captain said that he had never experienced this kind of emergency in his 19 years flying.
Passengers first noticed the smoke on takeoff and said soon after the fire alarms began to go off in the bathrooms.
Within about a minute the captain came on the intercom to say that although he wasn't sure what the problem was, there was an "issue" in the cabin and they would have to return to JFK.
Goldman said the banking of the plane to turn around was severe and caused babies on the plane to cry, which added to the stress of the situation.
Authorities said the source of the smoke was some leaking oil into the air conditioning system.
It was clear how long the delay would be before the passengers could make their way to the Bay Area.