I think I have figured out the issue.
On the site server under client installation there were no Accounts defined so that it can give site server privilleged access to install the client properly and then fetch policies.
This is what I have done:
In configmgr, goto Administration, Site Configuration, Sites. Then on the ribbon, click on Client Installation Settings and under Accounts tab add the domain admins or privilleged user who can install client on the machines.
After doing this now my windows 8.1 machines getting all the policies fetched and applied. I am still monitoring but I hope this is it.