Gills enjoyed plenty of chances in this match including a decent headed effort by Ian Cox that nearly saw the Gills open the scoring on the stroke of half time. Cox's header missed the target though and within a minute of the restart the Gills were behind.
Paul McLaren latched onto a Lee Williamson pass before slotting home past Brown.
Two goals in the final ten minutes by Deon Burton and Martin Butler were enough to secure a comfortable victory for Mick Harford's men.
Manager Neale Cooper made just one change to the side that drew with Brentford last week with Michael Flynn coming in for Brent Sancho.
Gillingham had to defend in numbers for the opening minutes of this game as Rotherham applied the early pressure.
John Mullin then saw an early shot deflected away from the target by Ian Cox before Brown denied Deon Burton with a decent one-handed save - it wouldn't have counted had it gone in though as the linesman flagged for offside.
Gills carved out an opportunity minutes later when Alan Pouton drove a 25-yard shot just wide of Neil Cutler's right hand post.
Then Darren Byfield saw a shot deflected wide before he headed a Danny Jackman cross straight into the arms of the Millers keeper.
Matty Jarvis shot just wide moments later before Brown pulled off a brilliant save to deny Williamson, who had struck from the edge of the area. Brown pushed the ball round the post to concede a corner kick, which Gills defended easily.
Further chances fell to both sides in the minutes that followed. Burton saw a speculative effort from distance go wide of the target before Darren Byfield, up the other end, shot narrowly wide twice.
The play switched from end to end as the first half drew to a close. Brown saved well to deny Martin Butler from the edge of the box, before Ian Cox missed the target with a free header from Danny Jackman's corner kick.
Rotherham stunned Gills in the first minute of the second half when they took the lead through Paul McLaren. Williamson found McLaren in space and he ran through on goal before slotting past Brown for 1-0.
Minutes later Jackman guided a cross into the arms of Cutler before Burton spun a shot wide of the Gills goal.
Gills were dealt a blow on 61 minutes when Chris Hope went off injured after a clash of heads. Paul Shields replaced him with Richard Rose moving to centre-half and Andrew Crofts dropping to right back.
Three minutes later Brown showed agility to deny Williamson before Cutler saved Pouton's long-range effort.
Further chances fell to the home side as they searched for a second goal. Burton fired wide on 72 minutes before Williamson shot wide from 20 yards moments later.
Gills fell further behind when Burton, who had been a thorn in Gills side for most of the afternoon, latched onto a cross from the right by Butler before smacking the ball past Brown from just inside the box.
Manager Neale Cooper immediately made a double substitution introducing Andy Hessenthaler and Steven Hislop for Jackman and Pouton.
Four minutes from time Cutler saved Michael Flynn's effort from distance before Butler put the result beyond doubt beating Brown from the edge of the penalty area with a left-footed strike.
Gills have just two days to pick themselves up from this disappointing defeat when they entertain Barnsley in the rearranged fixture on Tuesday night at Priestfield.
Rotherham: Cutler; Worrell; Barker; Gilchrist; Robertson; Mullin; McLaren; Williamson; Keane; Butler; Burton (Otsemobor, 87).
Subs: Montgomery; Vernazza; Campbell-Ryce; Hurst.
Gillingham: Brown; Jackman (Hislop, 82); Cox; Hope (Shields, 61); Rose; Crofts; Flynn; Jarvis; Pouton (Hessenthaler, 83); Harris; Byfield.
Subs: Crichton; Corneille.
Bookings: Pouton; Flynn (Gillingham); Gilchrist; Mullin (Rotherham)
Attendance: 4,253
Referee: Mr. A D'Urso
Report by: Simon Ongley














