It is important to realize that more than anything else your host institution feels responsible for your well being and are often afraid that something bad will happen. (We are all adults, but accidents do happen and people do feel responsible for us.) Additionally, a few years ago there was a serious car accident involving two JETs from Hyogo Prefecture. One of those JETs was killed. The other had a cracked skull. Because of events like these, some contracting organizations are wary of letting you drive.
As far as the technical question goes, CLAIR’s official policy is as follows: According to your contract, your school/BOE can tell you not to drive during work hours or commute by car to work, because you are their legal responsibility during that time. In your personal time, as long as you have a valid license and insurance, your school can’t tell you not to drive.
Note: There have been cases in the past where it was stipulated in a JET’s contract that he was not allowed to drive at all. This is extremely rare, but if this clause is in your contract and you signed the contract before coming to Japan, it unfortunately may be difficult to change.