Ñò
ÿ§Mc @ sm d d k Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k Z d d k l Z d d k l Z d d k
Z
e i i e i i
e i d ƒ ƒ Z e i d d d d d d
ƒ Z e ƒ Z d Z d „ Z e d
„ Z d „ Z d „ Z d „ Z d „ Z d d d „ Z d e d „ Z e e d e d „ Z d „ Z d „ Z d „ Z e! d j oe" e i ƒ d j o e ƒ nÕe" e i ƒ d j oBe i d d j o e ƒ q[e i d d j o e ƒ q[e i d i# d ƒ oŒ e i d d j o e d ƒ qÛe i d d! i$ d" ƒ Z% e" e% ƒ d j o% e e& e% d ƒ e& e% d ƒ ƒ qÛe e& e% d ƒ ƒ q[e i d d# j o e ƒ q[e i d d$ j o e ƒ q[e i d d% j o e ƒ q[n}e" e i ƒ d& j ofe i d d' j o e ƒ q[d( e i d j oY e i d i$ d( ƒ Z' xe( e& e' d ƒ e& e' d ƒ d ƒ D] Z) e i* e) ƒ qfWq[e i d i# d) ƒ o e e i d d ƒ q[e i d i# d* ƒ o e e i d d ƒ q[e i d i# d+ ƒ o e e i d d ƒ q[d" e i d j o e e i d ƒ q[e e& e i d ƒ d, e ƒn e i+ ƒ n d S(- iÿÿÿÿN( t Thread( t Queuei t usert roott passwdt guess8t dbt simplecdi c C sR t i t d ƒ } t | _ x/ t o' t i ƒ } t | | ƒ t i ƒ q Wd S( Ns /comment.sqlite3.db(
t sqlite3t connectt patht strt text_factoryt Truet qt gett fetchcmtt task_done( t connt topic( ( s" /var/www/simplecd.old/fcmt_noth.pyt thread_fetch s
c C s# t d } t | d ƒ i d | d ƒ d | } d GHt i | ƒ } t i d t i ƒ i | ƒ } t | ƒ } g } | o' t i d t i ƒ i | ƒ } | GHn | GH| o< x9 | D]- } t | d ƒ i | d ƒ t
i | ƒ q· Wn | o, | o% x" | D] } t | d t
ƒqWn d
S( s# search verycd, fetch search resultss /search.logt as
s% http://www.verycd.com/search/folders/s fetching search results ...s
/topics/(\d+)s /search/folders/(.*?\?start=\d+)t ,t fullN( R
t opent writet downloadt httpfetcht ret compilet DOTALLt findallt setR t putt searcht False( t keywordR t searchlogt urlt rest topicst linksR t key( ( s" /var/www/simplecd.old/fcmt_noth.pyR" $ s*
c C sÁ d } d GHt i | ƒ } t i d t i ƒ i | ƒ i ƒ } t i d t i ƒ i | ƒ } d } x> | D]6 } d G| d Gd GHt i | d ƒ | d | 7} qi Wt
t d
d ƒ i | ƒ d S(
s. read verycd hot res and keep update very day s http://www.verycd.com/s fetching homepage ...s çƒé—¨èµ„æº.*?s2 ]*>(《.*?》)[^<]*s. æ¯æ—¥çƒé—¨èµ„æº
s fetching hot topici s ...s5 %s
s /static/hot.htmlt wN(
R R R R R R" t groupR R R! R R
R ( R& t homet hotzonet hott htmlR ( ( s" /var/www/simplecd.old/fcmt_noth.pyR/ = s $ c
C sÝ d | j o: g } | i d ƒ D] } | t | ƒ q! ~ \ } } n t | ƒ } } x t | | d ƒ D]j } d | } t i | d t ƒ} t i d t i ƒ i | ƒ } | d GHx | D] } t
i | ƒ q¾ Wqk Wd S( s fetch normal res that need logint -i s, http://www.verycd.com/orz/page%d?stat=normalt needlogins
/topics/(\d+)i N( t splitt intt rangeR R R
R R R R R R! (
t pagest _[1]t xt ft tt pageR& t idxt idst id( ( s" /var/www/simplecd.old/fcmt_noth.pyt normalK s
:
c
C sÝ d | j o: g } | i d ƒ D] } | t | ƒ q! ~ \ } } n t | ƒ } } x t | | d ƒ D]j } d | } t i | d t ƒ} t i d t i ƒ i | ƒ } | d GHx | D] } t
i | ƒ q¾ Wqk Wd S( s! fetch request res that need loginR1 i s- http://www.verycd.com/orz/page%d?stat=requestR2 s
/topics/(\d+)i N( R3 R4 R5 R R R
R R R R R R! (
R6 R7 R8 R9 R: R; R& R< R= R> ( ( s" /var/www/simplecd.old/fcmt_noth.pyt requestY s
:
c C s€ d } d GHt i | ƒ } t i d t i ƒ i | ƒ } t | ƒ } | GHt i t i ƒ ƒ } x | D] } t
i | ƒ qe Wd S( s. read verycd feed and keep update very 30 min s http://www.verycd.com/sto/feeds fetching feed ...s
/topics/(\d+)N( R R R R R R R t timet mktimet gmtimeR R! ( R& t feedsR= t nowR> ( ( s" /var/www/simplecd.old/fcmt_noth.pyt feedg s i
i c C s× d } xÊ t | | d ƒ D]µ } d G| Gd GH| t | ƒ } t i | ƒ } t i d t i ƒ i | ƒ } | o | d } n q t i d t i ƒ i | ƒ } t | ƒ } | GHx | D] } t i
| ƒ q¸ Wq Wd S( Ns# http://www.verycd.com/sto/~all/pagei s
fetching lists ...s "topic-list"(.*?)"pnav"i s
/topics/(\d+)( R5 R R R R R R R R R R! ( t numt startt urlbaset iR& R' t res2R( R ( ( s" /var/www/simplecd.old/fcmt_noth.pyt updatey s
s 1-maxc C sD d } | i d ƒ oY t | i d ƒ d ƒ } t i | ƒ i ƒ } t t i d ƒ i | ƒ i d ƒ ƒ } n0 | i d ƒ } t | d ƒ } t | d ƒ } d G| Gd G| Gd GHxŠ t
| | d ƒ D]u } | d
| d } d G| Gd GHt i | ƒ } t i d
t i
ƒ i | ƒ } | GHx | D] }
t i |
ƒ q%WqÇ Wd S( Ns http://www.verycd.com/archives/t maxR1 i s archives/(\d+)i s fetching list fromt tos ...s %05ds .htmls
fetching froms
topics/(\d+)/( t endswithR4 R3 t urllibt urlopent readR R R" R, R5 R R R R R R! ( t rant debugRI t m1R' t m2t mRJ R& R= R> ( ( s" /var/www/simplecd.old/fcmt_noth.pyt fetchall‹ s& +
c C s^ d G| Gd GHd } | t | ƒ d t | ƒ } d } xA t d ƒ D]3 } y t i | d t d | ƒ} PWqD qD qD XqD W| d j oU t i d
ƒ i | ƒ } | o2 t | ƒ } x# | D] } | d j o qº qº WqÝ n t i d t i ƒ i | ƒ }
g } |
D]X } | | d i
d
d ƒ i ƒ | d i
d
d ƒ i ƒ | d i
d d ƒ i ƒ g q~ }
xÞ t t |
ƒ ƒ D]Ê }
t i d t i
ƒ i d |
|
d ƒ |
|
d ]*>([^<]*).*?(.*?).*?