Yes, you still need to file even if you have no payroll for the quarter. According to the IRS, any person or business that pays wages to an employee must file a Form 941 each quarter and must continue to do so even if there are no employees during some of the quarters.
Seasonal employers who aren't paying employee wages during one or more quarters, employers of household employees, and employers of agricultural employees are not required to file.