Over the past few years, there has been a tremendous increase in the interest and enthusiasm for sports among people. This has led to an increase in the importance given to video recording of various sports that capture even the minutest detail using high-end equipment. Recording and analysis have thereby become extremely crucial in sports like soccer that involve several complex and fast events. Ball detection and tracking along with player analysis have emerged as an area of interest among a lot of analysts and researchers. This is because it helps coaches in performance assessment of the team and in decision making to obtain optimized results. Video analysis can additionally be used by coaches and recruiters to look for new, talented players based on their previously played games. Ball detection also plays a pivotal role in assisting the referees in making decisions at game-changing moments. However, as the ball is almost always moving, its shape-appearance keeps changing over time and it is frequently occluded by players, it makes it difficult to track it throughout the game. We propose a deep learning-based YOLOv3 model for the ball and player detection in broadcast soccer videos. Initially, the videos are processed and unnecessary parts like zoom-ins, replays, etc., are removed to obtain only the relevant frames from each game. Tracking is achieved using the SORT algorithm which employs a Kalman filtering and bounding box overlap.