According to reports, scores of others were injured in the incident.
The landslide reportedly occurred close to a gurdwara around 5 pm when people who were sitting close to the edge of the road were crashed by a tree that toppled down along with debris.
The injured people were rushed to Jari Hospital for treatment.
The deceased reportedly included tourists.
Kullu Additional District Magistrate Ashwani Kumar told India Today that six people had lost their lives in the landslide, while many had been injured.
A rescue operation is currently going on.
Leader of Opposition in the Himachal Pradesh Assembly, Jairam Thakur, had condoled the loss of lives in the incident.
Kullu town is a major tourist attraction in the Himayalan state.