计算机毕业设计-基于Hadoop大数据技术的租房数据分析系统项目开发实战(附源码+论文)
租房数据分析系统是一个基于Hadoop的大数据平台,通过采用B/S架构,Flask框架以及MySQL数据库技术,旨在为用户提供全面的租房信息和数据支持。该系统具备丰富的功能,包括管理员端的系统首页、个人中心、用户管理、房屋信息管理、租房数据管理和系统管理,以及前台端的首页、房屋信息、租房数据、房屋资讯和个人中心等模块。通过这些功能,用户可以方便地查找合适的房源,了解租金走势和热门区域等信息,同时管
大家好!我是职场程序猿,感谢您阅读本文,欢迎一键三连哦。
精彩专栏推荐👇🏻👇🏻👇🏻
开发环境
- 开发语言:python
- 框架:Hadoop,Flask,Scrapy
- 数据库:mysql
- 数据库工具:Navicat
- 开发软件:PyCharm
- 浏览器:谷歌浏览器
演示视频
基于Hadoop的租房数据分析系统的设计与实现
论文目录
❤️【如需全文请按文末获取联系】
一、项目简介
租房数据分析系统是一个基于Hadoop的大数据平台,通过采用B/S架构,Flask框架以及MySQL数据库技术,旨在为用户提供全面的租房信息和数据支持。该系统具备丰富的功能,包括管理员端的系统首页、个人中心、用户管理、房屋信息管理、租房数据管理和系统管理,以及前台端的首页、房屋信息、租房数据、房屋资讯和个人中心等模块。通过这些功能,用户可以方便地查找合适的房源,了解租金走势和热门区域等信息,同时管理员可以对用户和房屋信息进行有效管理,保证系统的正常运行和数据的安全性。
二、系统设计
2.1软件功能模块设计
架构图是系统的体系结构,体系结构是体系结构体系的重要组成部分。租房数据分析推荐系统的总体结构设计如图4-2所示。
2.2数据库设计
在系统中将对 “用户、房屋信息、租房数据、房屋资讯、系统简介、收藏表”等几个主要的实体属性进行布局,如图4-3所示:
三、系统项目部分截图
3.1前台首页功能模块
当人们打开系统的网址后,首先看到的就是首页界面。在这里,人们能够看到租房数据分析系统的导航条,通过导航条导航进入各功能展示页面进行操作。系统首页界面如图5-1所示:
房屋信息:在房屋信息页面的输入栏中输入房屋名称和房屋地址进行查询,可以查看到房屋详细信息,并根据需要进行评论或收藏操作;房屋信息页面如图5-3所示:
3.2管理员模块实现
管理员登录界面图5-5所示。
管理员进入主页面,主要功能包括对系统首页、个人中心、用户管理、房屋信息管理、租房数据管理、系统管理等进行操作。管理员主页面如图5-6所示:
房屋信息管理界面如图5-8所示
租房数据管理界面如图5-9所示
在大数据分析方面,系统采用了Hadoop框架。Hadoop是一个能够处理大数据集的分布式存储和计算平台,它的核心是HDFS(Hadoop Distributed File System)和MapReduce计算模型。通过Hadoop,我们可以对收集到的大量数据进行存储和分析。看板页面如图5-10所示:
四、部分核心代码
# 注册接口
@main_bp.route("/python63we0i0p/zufang/register", methods=['POST'])
def python63we0i0p_zufang_register():
if request.method == 'POST':
msg = {'code': normal_code, 'message': 'success', 'data': [{}]}
req_dict = session.get("req_dict")
error = zufang.createbyreq(zufang, zufang, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = "注册用户已存在"
return jsonify(msg)
# 登录接口
@main_bp.route("/python63we0i0p/zufang/login", methods=['GET','POST'])
def python63we0i0p_zufang_login():
if request.method == 'GET' or request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
req_model = session.get("req_dict")
try:
del req_model['role']
except:
pass
datas = zufang.getbyparams(zufang, zufang, req_model)
if not datas:
msg['code'] = password_error_code
msg['msg']='密码错误或用户不存在'
return jsonify(msg)
req_dict['id'] = datas[0].get('id')
try:
del req_dict['mima']
except:
pass
return Auth.authenticate(Auth, zufang, req_dict)
# 登出接口
@main_bp.route("/python63we0i0p/zufang/logout", methods=['POST'])
def python63we0i0p_zufang_logout():
if request.method == 'POST':
msg = {
"msg": "退出成功",
"code": 0
}
req_dict = session.get("req_dict")
return jsonify(msg)
# 重置密码接口
@main_bp.route("/python63we0i0p/zufang/resetPass", methods=['POST'])
def python63we0i0p_zufang_resetpass():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success"}
req_dict = session.get("req_dict")
if req_dict.get('mima') != None:
req_dict['mima'] = '123456'
error = zufang.updatebyparams(zufang, zufang, req_dict)
if error != None:
msg['code'] = crud_error_code
msg['msg'] = error
else:
msg['msg'] = '密码已重置为:123456'
return jsonify(msg)
# 获取会话信息接口
@main_bp.route("/python63we0i0p/zufang/session", methods=['GET'])
def python63we0i0p_zufang_session():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "data": {}}
req_dict={"id":session.get('params').get("id")}
msg['data'] = zufang.getbyparams(zufang, zufang, req_dict)[0]
return jsonify(msg)
# 分类接口(后端)
@main_bp.route("/python63we0i0p/zufang/page", methods=['GET'])
def python63we0i0p_zufang_page():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data":{"currPage":1,"totalPage":1,"total":1,"pageSize":10,"list":[]}}
req_dict = session.get("req_dict")
userinfo = session.get("params")
try:
__hasMessage__=zufang.__hasMessage__
except:
__hasMessage__=None
if __hasMessage__ and __hasMessage__!="否":
tablename=session.get("tablename")
if tablename!="users" and session.get("params")!=None and zufang!='chat':
req_dict["userid"]=session.get("params").get("id")
tablename=session.get("tablename")
if tablename=="users" :
try:
pass
except:
pass
else:
mapping_str_to_object = {}
for model in Base_model._decl_class_registry.values():
if hasattr(model, '__tablename__'):
mapping_str_to_object[model.__tablename__] = model
try:
__isAdmin__=mapping_str_to_object[tablename].__isAdmin__
except:
__isAdmin__=None
try:
__authSeparate__ =mapping_str_to_object[tablename].__authSeparate__
except:
__authSeparate__ = None
if __isAdmin__!="是" and __authSeparate__ == "是" and session.get("params")!=None:
req_dict["userid"]=session.get("params").get("id")
else:
try:
del req_dict["userid"]
except:
pass
clause_args = []
or_clauses = or_(*clause_args)
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg['data']['total'], \
msg['data']['pageSize'] = zufang.page(zufang, zufang, req_dict, or_clauses)
return jsonify(msg)
# 排序接口
@main_bp.route("/python63we0i0p/zufang/autoSort", methods=['GET'])
def python63we0i0p_zufang_autosort():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data":{"currPage":1,"totalPage":1,"total":1,"pageSize":10,"list":[]}}
req_dict = session.get("req_dict")
req_dict['sort']='clicktime'
req_dict['order']='desc'
try:
__browseClick__= zufang.__browseClick__
except:
__browseClick__=None
if __browseClick__ =='是':
req_dict['sort']='clicknum'
elif __browseClick__ =='时长':
req_dict['sort']='browseduration'
else:
req_dict['sort']='clicktime'
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg['data']['total'], \
msg['data']['pageSize'] = zufang.page(zufang, zufang, req_dict)
return jsonify(msg)
# 分页接口(前端)
@main_bp.route("/python63we0i0p/zufang/list", methods=['GET'])
def python63we0i0p_zufang_list():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data":{"currPage":1,"totalPage":1,"total":1,"pageSize":10,"list":[]}}
req_dict = session.get("req_dict")
if req_dict.__contains__('vipread'):
del req_dict['vipread']
userinfo = session.get("params")
try:
__foreEndList__=zufang.__foreEndList__
except:
__foreEndList__=None
if __foreEndList__ and __foreEndList__!="否":
tablename=session.get("tablename")
if tablename!="users" and session.get("params")!=None:
req_dict['userid']=session.get("params").get("id")
try:
__foreEndListAuth__=zufang.__foreEndListAuth__
except:
__foreEndListAuth__=None
if __foreEndListAuth__ and __foreEndListAuth__!="否":
tablename=session.get("tablename")
if tablename!="users" and session.get("params")!=None:
req_dict['userid']=session.get("params").get("id")
tablename=session.get("tablename")
if tablename=="users" :
try:
del req_dict["userid"]
except:
pass
else:
mapping_str_to_object = {}
for model in Base_model._decl_class_registry.values():
if hasattr(model, '__tablename__'):
mapping_str_to_object[model.__tablename__] = model
try:
__isAdmin__=mapping_str_to_object[tablename].__isAdmin__
except:
__isAdmin__=None
if __isAdmin__!="是" and session.get("params")!=None:
req_dict["userid"]=session.get("params").get("id")
if 'luntan' in 'zufang':
if 'userid' in req_dict.keys():
del req_dict["userid"]
if 'discuss' in 'zufang':
if 'userid' in req_dict.keys():
del req_dict["userid"]
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg['data']['total'], \
msg['data']['pageSize'] = zufang.page(zufang, zufang, req_dict)
return jsonify(msg)
# 保存接口(后端)
@main_bp.route("/python63we0i0p/zufang/save", methods=['POST'])
def python63we0i0p_zufang_save():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
for key in req_dict:
if req_dict[key] == '':
req_dict[key] = None
error= zufang.createbyreq(zufang, zufang, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 添加接口(前端)
@main_bp.route("/python63we0i0p/zufang/add", methods=['POST'])
def python63we0i0p_zufang_add():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
try:
__foreEndListAuth__=zufang.__foreEndListAuth__
except:
__foreEndListAuth__=None
if __foreEndListAuth__ and __foreEndListAuth__!="否":
tablename=session.get("tablename")
if tablename!="users":
req_dict['userid']=session.get("params").get("id")
error= zufang.createbyreq(zufang, zufang, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 踩、赞接口
@main_bp.route("/python63we0i0p/zufang/thumbsup/<id_>", methods=['GET'])
def python63we0i0p_zufang_thumbsup(id_):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
id_=int(id_)
type_=int(req_dict.get("type",0))
rets=zufang.getbyid(zufang, zufang,id_)
update_dict={
"id":id_,
}
if type_==1:#赞
update_dict["thumbsupnum"]=int(rets[0].get('thumbsupnum'))+1
elif type_==2:#踩
update_dict["crazilynum"]=int(rets[0].get('crazilynum'))+1
error = zufang.updatebyparams(zufang, zufang, update_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 获取详情信息(后端)
@main_bp.route("/python63we0i0p/zufang/info/<id_>", methods=['GET'])
def python63we0i0p_zufang_info(id_):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
data = zufang.getbyid(zufang, zufang, int(id_))
if len(data)>0:
msg['data']=data[0]
#浏览点击次数
try:
__browseClick__= zufang.__browseClick__
except:
__browseClick__=None
if __browseClick__ and "clicknum" in zufang.__table__.columns:
click_dict={"id":int(id_),"clicknum":str(int(data[0].get("clicknum") or 0)+1)}
ret=zufang.updatebyparams(zufang,zufang,click_dict)
if ret!=None:
msg['code'] = crud_error_code
msg['msg'] = ret
return jsonify(msg)
# 获取详情信息(前端)
@main_bp.route("/python63we0i0p/zufang/detail/<id_>", methods=['GET'])
def python63we0i0p_zufang_detail(id_):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
data = zufang.getbyid(zufang, zufang, int(id_))
if len(data)>0:
msg['data']=data[0]
#浏览点击次数
try:
__browseClick__= zufang.__browseClick__
except:
__browseClick__=None
if __browseClick__ and "clicknum" in zufang.__table__.columns:
click_dict={"id":int(id_),"clicknum":str(int(data[0].get("clicknum") or 0)+1)}
ret=zufang.updatebyparams(zufang,zufang,click_dict)
if ret!=None:
msg['code'] = crud_error_code
msg['msg'] = ret
return jsonify(msg)
# 更新接口
@main_bp.route("/python63we0i0p/zufang/update", methods=['POST'])
def python63we0i0p_zufang_update():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
if req_dict.get("mima") and "mima" not in zufang.__table__.columns :
del req_dict["mima"]
if req_dict.get("password") and "password" not in zufang.__table__.columns :
del req_dict["password"]
try:
del req_dict["clicknum"]
except:
pass
error = zufang.updatebyparams(zufang, zufang, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 删除接口
@main_bp.route("/python63we0i0p/zufang/delete", methods=['POST'])
def python63we0i0p_zufang_delete():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
error=zufang.delete(
zufang,
req_dict
)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 投票接口
@main_bp.route("/python63we0i0p/zufang/vote/<int:id_>", methods=['POST'])
def python63we0i0p_zufang_vote(id_):
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success"}
data= zufang.getbyid(zufang, zufang, int(id_))
for i in data:
votenum=i.get('votenum')
if votenum!=None:
params={"id":int(id_),"votenum":votenum+1}
error=zufang.updatebyparams(zufang,zufang,params)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 分组统计接口
@main_bp.route("/python63we0i0p/zufang/group/<columnName>", methods=['GET'])
def python63we0i0p_zufang_group(columnName):
'''
分组统计接口
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
userinfo = session.get("params")
msg['data'] = zufang.groupbycolumnname(zufang,zufang,columnName,req_dict)
msg['data'] = msg['data'][:10]
msg['data'] = [ {**i,columnName:str(i[columnName])} if columnName in i else i for i in msg['data']]
json_filename='zufang'+f'_group_{columnName}.json'
where = ' where 1 = 1 '
sql = "SELECT COUNT(*) AS total, " + columnName + " FROM zufang " + where + " GROUP BY " + columnName
with open(json_filename, 'w', encoding='utf-8') as f:
f.write(json.dumps(msg['data'], indent=4, ensure_ascii=False))
app.executor.submit(upload_to_hdfs, json_filename)
app.executor.submit(MRMySQLAvg.run)
return jsonify(msg)
# 按值统计接口
@main_bp.route("/python63we0i0p/zufang/value/<xColumnName>/<yColumnName>", methods=['GET'])
def python63we0i0p_zufang_value(xColumnName, yColumnName):
'''
按值统计接口,
{
"code": 0,
"data": [
{
"total": 10.0,
"shangpinleibie": "aa"
},
{
"total": 20.0,
"shangpinleibie": "bb"
},
{
"total": 15.0,
"shangpinleibie": "cc"
}
]
}
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
userinfo = session.get("params")
msg['data'] = zufang.getvaluebyxycolumnname(zufang,zufang,xColumnName,yColumnName,req_dict)
msg['data'] = msg['data'][:10]
return jsonify(msg)
# 按日期统计接口
@main_bp.route("/python63we0i0p/zufang/value/<xColumnName>/<yColumnName>/<timeStatType>", methods=['GET'])
def python63we0i0p_zufang_value_riqi(xColumnName, yColumnName, timeStatType):
'''
按日期统计接口
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
userinfo = session.get("params")
where = ' where 1 = 1 '
sql = ''
if timeStatType == '日':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m-%d') {0}, sum({1}) total FROM zufang {2} GROUP BY DATE_FORMAT({0}, '%Y-%m-%d')".format(xColumnName, yColumnName, where, '%Y-%m-%d')
if timeStatType == '月':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m') {0}, sum({1}) total FROM zufang {2} GROUP BY DATE_FORMAT({0}, '%Y-%m')".format(xColumnName, yColumnName, where, '%Y-%m')
if timeStatType == '年':
sql = "SELECT DATE_FORMAT({0}, '%Y') {0}, sum({1}) total FROM zufang {2} GROUP BY DATE_FORMAT({0}, '%Y')".format(xColumnName, yColumnName, where, '%Y')
data = db.session.execute(sql)
data = data.fetchall()
results = []
for i in range(len(data)):
result = {
xColumnName: decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
results.append(result)
msg['data'] = results
json_filename='zufang'+f'_value_{xColumnName}_{yColumnName}.json'
with open(json_filename, 'w', encoding='utf-8') as f:
f.write(json.dumps(results, indent=4, ensure_ascii=False))
app.executor.submit(upload_to_hdfs, json_filename)
app.executor.submit(MRMySQLAvg.run)
return jsonify(msg)
# 按值统计(多)
@main_bp.route("/python63we0i0p/zufang/valueMul/<xColumnName>", methods=['GET'])
def python63we0i0p_zufang_valueMul(xColumnName):
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
req_dict = session.get("req_dict")
userinfo = session.get("params")
where = ' where 1 = 1 '
for item in req_dict['yColumnNameMul'].split(','):
sql = "SELECT {0}, sum({1}) AS total FROM zufang {2} GROUP BY {0} LIMIT 10".format(xColumnName, item, where)
L = []
data = db.session.execute(sql)
data = data.fetchall()
for i in range(len(data)):
result = {
xColumnName: decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
L.append(result)
msg['data'].append(L)
return jsonify(msg)
# 按值统计(多)
@main_bp.route("/python63we0i0p/zufang/valueMul/<xColumnName>/<timeStatType>", methods=['GET'])
def python63we0i0p_zufang_valueMul_time(xColumnName):
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
req_dict = session.get("req_dict")
userinfo = session.get("params")
timeStatType = req_dict['timeStatType']
where = ' where 1 = 1 '
for item in req_dict['yColumnNameMul'].split(','):
sql = ''
if timeStatType == '日':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m-%d') {0}, sum({1}) total FROM zufang {2} GROUP BY DATE_FORMAT({0}, '%Y-%m-%d') LIMIT 10".format(xColumnName, item, where, '%Y-%m-%d')
if timeStatType == '月':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m') {0}, sum({1}) total FROM zufang {2} GROUP BY DATE_FORMAT({0}, '%Y-%m') LIMIT 10".format(xColumnName, item, where, '%Y-%m')
if timeStatType == '年':
sql = "SELECT DATE_FORMAT({0}, '%Y') {0}, sum({1}) total FROM zufang {2} GROUP BY DATE_FORMAT({0}, '%Y') LIMIT 10".format(xColumnName, item, where, '%Y')
L = []
data = db.session.execute(sql)
data = data.fetchall()
for i in range(len(data)):
result = {
xColumnName: decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
L.append(result)
msg['data'].append(L)
return jsonify(msg)
# 总数量
@main_bp.route("/python63we0i0p/zufang/count", methods=['GET'])
def python63we0i0p_zufang_count():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": 0}
req_dict = session.get("req_dict")
userinfo = session.get("params")
msg['data'] = zufang.count(zufang, zufang, req_dict)
return jsonify(msg)
# 统计接口
@main_bp.route("/python63we0i0p/zufang/remind/<columnName>/<type>", methods=['GET']) #
def python63we0i0p_zufang_remind(columnName,type):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, 'count': 0}
# 组合查询参数
params = session.get("req_dict")
remindstart = 0
remindend =9999990
if int(type)==1:#数字
if params.get('remindstart') == None and params.get('remindend') != None:
remindstart = 0
remindend = int(params['remindend'])
elif params.get('remindstart') != None and params.get('remindend') == None:
remindstart = int(params['remindstart'])
remindend = 999999
elif params.get('remindstart') == None and params.get('remindend') == None:
remindstart = 0
remindend = 999999
else:
remindstart = params.get('remindstart')
remindend = params.get('remindend')
elif int(type)==2:#日期
current_time=int(time.time())
if params.get('remindstart') == None and params.get('remindend') != None:
starttime=current_time-60*60*24*365*2
params['remindstart'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(starttime))
endtime=current_time+60*60*24*params.get('remindend')
params['remindend'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(endtime))
elif params.get('remindstart') != None and params.get('remindend') == None:
starttime= current_time - 60 * 60 * 24 * params.get('remindstart')
params['remindstart']=time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(starttime))
endtime=current_time+60*60*24*365*2
params['remindend'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(endtime))
elif params.get('remindstart') == None and params.get('remindend') == None:
starttime = current_time - 60 * 60 * 24 * 365 * 2
params['remindstart'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(starttime))
endtime = current_time + 60 * 60 * 24 * 365 * 2
params['remindend'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(endtime))
data = zufang.getbetweenparams(
zufang,
zufang,
columnName,
{
"remindStart": remindstart,
"remindEnd": remindend
}
)
msg['count'] = len(data)
return jsonify(msg)
#分类列表
@main_bp.route("/python63we0i0p/zufang/lists", methods=['GET'])
def python63we0i0p_zufang_lists():
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
list,_,_,_,_ = zufang.page(zufang,zufang,{})
msg['data'] = list
return jsonify(msg)
获取源码或论文
如需对应的论文或源码,以及其他定制需求,也可以下方微信联系我。
更多推荐
所有评论(0)