Public works minister Shivpal Yadav made the remarks while speaking to officials at Etah district.
Mr Yadav later said he was against corruption and that news channels had shown an edited version of his remarks.
The remarks have caused an uproar in a country which is witnessing a mounting campaign against corruption.
"As long as you work hard and get things done, you can steal a little. But this should not translate into robbery," Mr Yadav was shown on news channels as telling officials at a meeting.
Mr Yadav, who belongs to the governing Samajwadi Party, later blamed the media for filming the meeting without permission.
"Only an edited clip of my comments at the meeting is being shown on the news channels. I was actually discussing ways to check corruption at a meeting with district officials. I am resolutely against corruption", he told reporters.
Mr Yadav said his government had been elected on a pledge to curb "rampant" corruption under the previous government led by the Bahujan Samaj Party.
He said he had taken action against a number of corrupt officials, and it was wrong to think he was condoning graft.
A recent survey said corruption in India had cost billions of dollars and threatened to derail growth.
Note: At least our ministers are not that dumb, they don't go public saying stuff like "it's OK to rape a couple of teenagers once in a while"...or "it's OK to collect bribes from large construction companies" etc...