Your attorney is incorrect. There are no time or place restrictions on a Georgia limited permit. The only limitation is that you must be able to justify that you are driving to or from or for work.
As always, Mr. Hawkins is correct. As long as you are driving for work (not headed to the grocery store or running errands before or after work or on lunch hour) but actually travelling between work and home or for work purposes, then you should be ok. Try to always have some sort of documentation showing where you are headed and why if at all possible.
To add to the above correct answers, you are also allowed to travel for medical reasons, to school and to court-ordered meetings, such as DUI school or AA. You are also able to take your children to school and / or medical care.
Sign up to receive a 3-part series of useful information and legal advice about DUIs.