千萬級別資料的匯出到excel實現(以自己以前做的訂單匯出為demo給大家參考)
阿新 • • 發佈:2018-12-16
考慮幾個重點:
1,伺服器承載 2,redis資料快取避免資料重複匯出,3,匯出後的資料處理 4,死迴圈 5,資料大小,限制大變數的出現
遇到這樣的需求,大家根據自己的需求去處理業務,多方位去考慮程式的可執行性,效能等多方面因素(儘量減少迴圈中的查詢次數)
不多說附上程式碼供大家參考
<?php class ReportOrdersExport extends Job implements SelfHandling, ShouldQueue { protected $start; protected $end; protected $shops; protected $email; protected $shopArr; protected $productArr; protected $brandArr; protected $catArr; /** * ReportOrdersExport constructor. * @param $start * @param $end * @param $email * @param $shops */ public function __construct($start, $end, $email, $shops) { $this->start = $start; $this->end = $end; $this->shops = $shops; $this->email = $email; } /** * Execute the job. * * @return void */ public function handle() { $start = $this->start; $end = $this->end; $shops = $this->shops; $params['start'] = $start; $params['end'] = $end; $params['shops'] = []; $params['email'] = $this->email; $key = 'report_orders_'.$start.$end.implode('', $shops); if (0 < count($shops)) $params['shops'] = Shops::whereIn('shop_id', $shops)->lists('shop_code')->toArray(); $file = Cache::get($key); if ($file) return $this->sendEmail($params, $file); $this->setLoading(); //檔名稱 $file = sprintf('%s/%s/%s/%s', date('Y'), date('m'), date('d'), time().'.csv'); //根據自己的需求匯出對應的資料 $title = "訂單號,會員ID,下單時間,訂單總額,顏色,尺碼,頁面價格,一口價,實付金額,商品數量...."; Storage::append($file, iconv('utf-8', 'gb2312', $title)); //死迴圈(我是以店鋪為單位去匹配對應的資料,然後匯出) $step = 100; for ($k = 0; 1 == 1; $k += $step) { $model = Orders::where('status', '<>', 'dead') ->where('pay_status', '1') ->where('pay_time', '>=', $start) ->where('pay_time', '<', $end); if (0 < count($shops)) $model->whereIn('shop_id', $shops); $model->orderBy('pay_time', 'asc'); $model->skip($k); $model->take($step); $rows = $model->get(['order_id', 'final_amount', 'member_id', 'lv_name', 'shop_id', 'createtime', 'received_time', 'payment', 'status', 'tostr', 'source', 'cost_item', 'pmt_order', 'discount', 'pay_time', 'cost_freight', 'pay_status', 'ship_addr', 'ship_name', 'ship_mobile', 'ship_area']); //表示已經沒有資料 if (1 > count($rows)) break; foreach ($rows as $row) { $text = $row->order_id.','; $member = Members::select(['regtime', 'member_lv_id', 'login_account', 'order_num'])->where('member_id', $row->member_id)->first(); $text .= $row->member_id.','; $text .= ($member ? $member->login_account : '').','; $text .= ($member ? $member->order_num : '').','; $text .= ($row->lv_name ? ($member ? $this->getMemberLvName($member->member_lv_id) : '') : '').','; $text .= ($member ? date('Y-m-d H:i:s', $member->regtime) : '').','; $text .= ($row->shop_id ? $this->getShopCode($row->shop_id) : '').','; $text .= date('Y-m-d H:i:s', $row->createtime).','; $items = OrderItems::where('order_id', $row->order_id)->get(['goods_id', 'product_id', 'bn', 'name', 'g_price', 'price', 'nums']); //特殊情況下沒明細?應該不會出現 if (1 > count($items)) { Storage::append($file, $text."\n"); } $i = 0; foreach ($items as $item) { if ($i > 0) $text = ',,,,,,,,,,,,,,,,,,,,,'; $product = $this->getProductInfo($item->product_id); $text .= ($product ? $product->goods_sn : '').','; $text .= $item->bn.','; $text .= $item->name.','; $text .= ($product ? $this->getBrandName($product->brand_id) : '').','; $text .= ($product ? $this->getCatInfo($product->cat_id)[2] : '').','; Storage::append($file, iconv('utf-8', 'gb2312//TRANSLIT//IGNOR', str_replace("\n", "", $text))); $i++; } } } //同1份檔案8小時內不可以重複匯出 Cache::put($key, $file, 60 * 8); $this->removeLoading(); return $this->sendEmail($params, $file); } protected function sendEmail($params, $file) { return Mail::send('emails.report', $params, function ($message) use ($params, $file) { $to = [$params['email']]; $message->to($to)->subject('系統郵件::訂單資料'); $message->attach(storage_path("app/$file")); }); } protected function setLoading() { return Cache::put('report_orders_loading', 1, 60 * 24); } protected function removeLoading() { return Cache::forget('report_orders_loading'); } }