Reports indicate that a minority of all confirmed COVID-19 patients required hospitalization, and continuous monitoring of these patients is critical in the management of the disease. Severe illness and death in COVID-19 patients are often due to the development of interstitial pneumonia. The lung function could deteriorate rapidly in severe COVID-19 patients who, in turn, require urgent admission to an intensive care unit (ICU) for mechanical ventilation to maintain lung function until they recover. With limited ICU facilities in many hospitals and rural areas, detection of COVID-19 pneumonia using non-invasive imaging could play a crucial role in the effective management of patients.  Ultrasound imaging offers a solution to detect pneumonia in COVID-19 patients, and it is inexpensive, portable, and free of ionization radiation. In addition, wireless or hand-held scanners are relatively easy to disinfect due to their small size. Despite these advantages, ultrasound imaging is notoriously difficult to interpret and often requires an opinion from an expert. In this study, we proposed a deep convolutional neural network approach based on the Kinetics-I3D to automatically detect different ultrasound imaging based indicators such as A-lines, B-lines, and consolidation. The detection results were compared against ground truth annotations by expert radiologists. This study demonstrated that there is a very good agreement between the proposed automated method and the annotations by radiologists. We believe that the proposed automated analysis of portable ultrasound images can help triage patients presented to emergency with flu-like symptoms, determine who needs to be hospitalized, and predict patients who require ICU admission.