XylotrechusZ
�
t��g` � �� � d Z ddlZddlmZ ddlmZmZmZmZ ddl Z ddl m
Z
m Z m Z m
Z
mZmZ ddlmZmZ e G d� d� � � � Zd ee fd
�Zd � Zd eeee f fd �ZdS )
zF
Represents storage where ssa data is collected, stored and extracted
� N)� dataclass)�List�Iterator�Tuple�Dict)�func�cast�case�literal_column�distinct�text)�
session_scope�
RequestResultc �D � e Zd ZU eed< ee ed< eed< eed< dS )�
DomainData� domain_name�domain_total_reqs�is_a_wordpress_domain� urls_numberN)�__name__�
__module__� __qualname__�str�__annotations__r �int�bool� � �J/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/storage.pyr r sF � � � � � � ������C�y� � � � � � � ������r r �returnc # �z K � t | � � 5 }|� t j t j dt j � � t j t j � � t j t j
� � t j t t j
� � � � � � � t j t j dt j � � � � � t j t j dt j � � � � }t! j |d� �� � }|D ]^\ }}t% |� � }d}dgdz }|D ]*\ } }
} } }
| |t' |
� � < t ||
� � }�+t) ||| |�� � V � �_ ddd� � dS # 1 swxY w Y dS )z7
Iterates data from database domain-by-domain.
�%Hc � � | d S �Nr r ��items r �<lambda>z#iter_domains_data.<locals>.<lambda>/ s
� �VZ�[\�V]� r )�keyr � )r r r r N)r �queryr �domainr �strftime�
created_at�Count�id�max� wordpress�countr �path�group_by�order_by� itertools�groupby�tupler r )�engine�db�results_by_hour�results_by_hour_groupedr �group�domain_results_by_hourr �requests_number_by_hour�_�hour� requests_num� is_wordpress�urlss r �iter_domains_datarE s� � � � �
�v� � � !�"� �(�(� � � �M�$�
� 8� 9� 9� �J�}�'� (� (� �H�]�,� -� -� �J�x�
� 2�3�3� 4� 4�
�
� �(� � � �M�$�
� 8� 9� 9�
�
� �(� � � �M�$�
� 8� 9� 9�
�
� � #,�"3�O�I]�I]�"^�"^�"^� �"9� � � �K��%*�5�\�\� "��K� ()�c�B�h� #�=S�
5�
5�9��4� �|�T�5A�'��D� � �2� !�+�t� 4� 4� � � � '�"9�&2� '� � � �
�
�
�
� �#!� !� !� !� !� !� !� !� !� !� !� !���� !� !� !� !� !� !s �FF0�0F4�7F4c
# � K � t | � � 5 }d� |D � � }|� t j t j dt j � � t j t t j t j � � � � � d� � t j
t j � � � d� � t j t t j t j � � � � � d� � � � � t j |k � � � t% t j � |� � j � ddi�� � j � � � � � t j t j dt j � � � � � t j t j dt j � � � � }d }d
gd z d
gd z d
gd z } }}|D ]q\ }
} } }
}|r/||
k r)|t3 |||| � � � fV � d
gd z d
gd z d
gd z } }}| |t5 | � � < |
|t5 | � � < || t5 | � � < |
}�r|
t3 |
||| � � � fV � d d d � � d S # 1 swxY w Y d S )
z6
Iterates urls data from database url-by-url.
c �: � g | ]}|� d d� � ��S )�:z\:)�replace)�.0r3 s r �
<listcomp>z"iter_urls_data.<locals>.<listcomp>H s& � � L� L� L�$�T�\�\�#�u� 5� 5� L� L� Lr r" �url_throttled_reqs�url_total_reqs�
url_slow_reqs�
literal_bindsT)�compile_kwargsNr r) )r3 rL rM rN )r r* r r3 r r, r- �Sumr �hitting_limits�
sqlalchemy�Integer�labelr. r/ �is_slow_request�filterr+ r
�in_�
expression�compile�stringr4 r5 �dictr )r9 r � all_pathsr: �all_paths_escaped� urls_data�
previous_pathrL rM rN r3 rA �url_throttled_req�
url_total_req� url_slow_reqs r �iter_urls_datard C s � � � �
�v� � � 2
�"� L� L�)� L� L� L���H�H� � � �M�$�
� 8� 9� 9� �H�T� �,�j�.@�� �
�
��u�)�*�*� �J� � �
�
��e�$�%�%� �H�T� �-�z�/A�C� C�
�
��e�O�$�$�
�
� �&� � �K� /�
�
�
�&� � �#�'�'�(9�:�:�E�M�M�^m�os�]t�M�u�u�|� }� }�
�
�
�(� � ��
�d�M�4L� M� M�
�
�
�(� � ��
�d�M�4L� M� M�
�
�% �, �
�
�C�"�H�q�c�B�h���b�� -:�N�� KT� !� !� F�D�$�)�=�,� �
1� �$�!6�!6�#�T�&�'9�#1�"/� &� &� &� � � � � �C�"�H�q�c�B�h���b�� 5B�N�"� ->� �s�4�y�y� )�(5�N�3�t�9�9� %�'3�M�#�d�)�)� $� �M�M��D�� 1�)�'�
�
�
�
�
�
�
�[2
� 2
� 2
� 2
� 2
� 2
� 2
� 2
� 2
� 2
� 2
� 2
���� 2
� 2
� 2
� 2
� 2
� 2
s �J J<�<K �K c # � K � t | � � 5 }|� t j t j � � � t j |k � � � t j � � }t j t |� � d� � � }|D ]\ }}|d� |D � � fV � � ddd� � dS # 1 swxY w Y dS )zA
Get information about durations of requests url-by-url.
c � � | d S r$ r r% s r r' z#get_url_durations.<locals>.<lambda>� s
� �$�q�'� r c � � g | ]\ }}|��S r r )rJ r@ �durations r rK z%get_url_durations.<locals>.<listcomp>� s � �:�:�:�[�Q���:�:�:r N) r r* r r3 rh rW r+ r5 r6 r7 �list)r9 r r: r_ �durations_by_pathr( r= s r �get_url_durationsrk | s) � � � �
�v� � �
;�"��H�H� � � � "�
�
� �&� � �K� /�
�
�
�(� � �
�
� � &� -� ��O�O� 1� 1� 3� 3��+� ;� ;�J�C���:�:�E�:�:�:�:� :� :� :� :� ;�
;�
;�
;�
;�
;�
;�
;�
;�
;�
;�
;�
;����
;�
;�
;�
;�
;�
;s �B$C�C� C)�__doc__r6 � dataclassesr �typingr r r r rS r r r
r r r
�ssa.dbr r r rE rd r r rk r r r �<module>rp s> ��� � � � � � !� !� !� !� !� !� .� .� .� .� .� .� .� .� .� .� .� .� � � � � G� G� G� G� G� G� G� G� G� G� G� G� G� G� G� G� /� /� /� /� /� /� /� /� �� � � � � � � ���%��*�!5� %� %� %� %�N6
� 6
� 6
�r;�d�3��c�
�?�.C� ;� ;� ;� ;� ;� ;r